Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Computation of compression parameters via OpenVINO models #2727

Merged
merged 109 commits into from
Jan 23, 2025

Conversation

nikita-savelyevv
Copy link
Collaborator

@nikita-savelyevv nikita-savelyevv commented Jun 11, 2024

Changes

  • Implemented OpenVINO model graphs which are used for calculation of compressed and decompressed weights. Since these models are compiled, calculation become significantly faster especially for larger models and int4 compression.
  • This functionality is exposed by two methods at weight_lowering.py:
    • do_int_quantization() is used for computing a compressed weight. Possible signatures:
      • weight -> compressed_weight, scale, (zero_point for asymmetric compression)
      • weight, scale, (zero_point) -> compressed_weight, scale, (zero_point)
    • calculate_quantized_dequantized_weight() is used for computing a decompressed weight. Possible signatures:
      • weight -> decompressed_weight
      • weight, scale, (zero_point) -> decompressed_weight
      • weight -> decompressed_weight, compressed_weight, scale, (zero_point)
      • weight, scale, (zero_point) -> decompressed_weight, compressed_weight, scale, (zero_point)
    • Output scale and zero_point are the same as the ones given as input (if they were given at all).
    • Computation is done via OV models only if openvino package is installed and input tensors are not torch tensors.
  • Introduce a new NNCF Tensor backend for storing instances of openvino.Tensor. Implementation for this backend is limited by only the required functionality, e.g. addition of OV Tensors is not supported because it is not needed.
    • Introduction of OV Tensors is required for seamless handling of tensors in bf16, u4 and i4 data types. For example, bf16 constants are read from an OpenVINO LLM and given as inputs to a compressing OpenVINO model. u4 and i4 compressed weights are seamlessly inserted into the resulting compressed OpenVINO model.
    • Added as_numpy_tensor() method to convert an NNCF Tensor to numpy backend. Currently only OV -> NP conversion is required.
  • All calculations are aligned with reference numpy implementation. Some performance and memory sacrifices had to be made for such alignment.

Data-free asymmetric compression:
image

Data-free symmetric compression:
image

Data-aware compression:
image

Reason for changes

Reducing model compression time. Only OpenVINO model compression backend is affected.

Related tickets

139047

Tests

  • tests/openvino/native/quantization/test_ov_modeling_compression.py::test_quantization_alignment -- check aligment with reference numpy implementation
  • tests/openvino/native/test_openvino_modeling.py -- checks OV modeling framework hyperparameters
  • tests/openvino/native/test_tensor.py -- NNCF OV Tensor backend tests

Validation jobs:

@github-actions github-actions bot added NNCF Common Pull request that updates NNCF Common NNCF OpenVINO Pull requests that updates NNCF OpenVINO NNCF PTQ Pull requests that updates NNCF PTQ labels Jun 11, 2024
@nikita-savelyevv nikita-savelyevv force-pushed the compress-via-openvino branch 4 times, most recently from 55cafaa to a68a63d Compare July 3, 2024 18:31
@nikita-savelyevv nikita-savelyevv force-pushed the compress-via-openvino branch 4 times, most recently from 6b98ddd to 3d9faa4 Compare July 16, 2024 14:19
@nikita-savelyevv nikita-savelyevv force-pushed the compress-via-openvino branch 6 times, most recently from 1c85732 to b527cac Compare September 6, 2024 11:11
@github-actions github-actions bot added the documentation Improvements or additions to documentation label Sep 6, 2024
@nikita-savelyevv nikita-savelyevv force-pushed the compress-via-openvino branch 2 times, most recently from ac3ea02 to 2a3a63c Compare September 11, 2024 12:59
@nikita-savelyevv nikita-savelyevv force-pushed the compress-via-openvino branch 2 times, most recently from fe30c13 to 19ea412 Compare October 21, 2024 08:52
@nikita-savelyevv nikita-savelyevv force-pushed the compress-via-openvino branch 3 times, most recently from eef34f8 to ca3447c Compare October 26, 2024 13:40

Verified

This commit was created on GitHub.com and signed with GitHub’s verified signature.
@github-actions github-actions bot added the NNCF PT Pull requests that updates NNCF PyTorch label Jan 16, 2025
nncf/common/utils/decorators.py Outdated Show resolved Hide resolved
nncf/common/utils/decorators.py Outdated Show resolved Hide resolved
nncf/common/utils/decorators.py Outdated Show resolved Hide resolved
nncf/common/utils/decorators.py Outdated Show resolved Hide resolved
nncf/common/utils/decorators.py Outdated Show resolved Hide resolved
nncf/openvino/graph/node_utils.py Show resolved Hide resolved
nncf/tensor/functions/numeric.py Outdated Show resolved Hide resolved
nncf/tensor/functions/ov_numeric.py Outdated Show resolved Hide resolved
nncf/torch/quantization/algo.py Outdated Show resolved Hide resolved
nncf/tensor/functions/ov_numeric.py Outdated Show resolved Hide resolved
nncf/tensor/functions/ov_numeric.py Outdated Show resolved Hide resolved
nncf/common/logging/logger.py Outdated Show resolved Hide resolved
nncf/common/utils/caching.py Outdated Show resolved Hide resolved
nncf/common/logging/logger.py Show resolved Hide resolved
nncf/common/utils/caching.py Outdated Show resolved Hide resolved
Copy link
Contributor

@alexsu52 alexsu52 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

  1. Please address minor comments from the last round of review.
  2. The comment Computation of compression parameters via OpenVINO models #2727 (comment) will be addressed int the follow-up PR.

nncf/common/utils/caching.py Outdated Show resolved Hide resolved
@alexsu52 alexsu52 merged commit f3f232f into openvinotoolkit:develop Jan 23, 2025
17 checks passed
alexsu52 pushed a commit that referenced this pull request Jan 24, 2025
### Changes

Follow up to #2727

1. Do not use `infer_request.results`
2. Replace `>=` with `opset.greater_equal()`
3. Rename `ov_numeric.py` to `openvino_numeric.py`

### Reason for changes

1. Improve int4 compression time by up to ~10%
2. Avoid warning: `DeprecationWarning: greater_equal is deprecated and
will be removed in version 2025.3. Use ops.greater_equal instead`
3. Fix onnx install test

### Related tickets

139047

### Tests

- https://github.com/openvinotoolkit/nncf/actions/runs/12947249537
- NNCF/job/manual/job/post_training_weight_compression/301/
- NNCF/job/nightly/job/test_examples/653/
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
documentation Improvements or additions to documentation NNCF Common Pull request that updates NNCF Common NNCF OpenVINO Pull requests that updates NNCF OpenVINO NNCF PT Pull requests that updates NNCF PyTorch NNCF PTQ Pull requests that updates NNCF PTQ NNCF TF Pull requests that updates NNCF TensorFlow
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants